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Anomalous Wtb couplings modify the angular correlations of the top quark decay products and 
change the single top quark production cross section. We present limits on anomalous top quark 
couplings by combining information from W boson helicity measurements in top quark decays and 
anomalous coupling searches in the single top quark final state. We set limits on right-handed vector 
couplings as well as left-handed and right-handed tensor couplings based on about 1 fb _1 of data 
collected by the DO experiment. 

PACS numbers: 14.65.Ha; 12.15.Ji; 13.85.Qk 



The top quark is by far the heaviest fermion in the standard model (SM), and thus has the strongest 
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coupling to the Higgs boson of all SM fermions. This 
makes the top quark and its interactions an ideal place 
to look for new physics related to electroweak symmetry 
breaking. The coupling between the top quark and the 
weak gauge bosons may be altered by physics beyond the 
SM. In particular the coupling between the top quark 
and the W boson determines most of the top quark 
phenomenology and can be sensitively probed at hadron 
colliders [l[ ■ The effective Lagrangian describing the Wtb 
interaction including operators up to dimension five is Q : 



C = 



-JLbrfVtbitfPL 

9 i ^ v 1vV tb lfL 
V2 M w 



f?PR)tW- 



cases, pairing the left-handed vector coupling form factor 
fi with each of the other three form factors. We refer 
to these as (Li,Ri), (L\,L 2 ), and (Li,R 2 ). For each 
pair of form factors a likelihood distribution is extracted 
from the W helicity measurement of the decay angle 
distribution in top quark decays. All top quark pair 
events with decays to at least one lepton (electron or 
muon) arc included in the W helicity measurement. 
This likelihood is then combined with the result of the 
anomalous couplings search in the single top quark final 
state in a Baycsian statistical analysis, yielding a two- 
dimensional posterior probability density as a function 
of both form factors. We extract limits on f^, f£, and 
{J^Pl + f^PjdtWp + h.c. , (1) f 2 by projecting the two-dimensional posterior onto the 

corresponding form factor axis. 

The W boson helicity measurement, described in 
Ref. Q, uses events in both the £+jets (ti — » 
W + W~bb -> tvqq'bb) and dilepton (ti -> W+W-bb -> 
tvl'v'bb) final states. The measurement variable is 9*, 
the angle between the down-type fermion and top quark 
momenta in the W boson rest frame. To evaluate this 
variable, we assign a momentum to the neutrino(s) cither 
via a constrained kinematic fit (in the £+jets channel) or 
an algebraic solution (in the dilepton channel). 

We use the ALPGEN leading-order Monte Carlo (MC) 
event generator [{|, interfaced to PYTHIA [r|, to model 
ti events as well as jy+jets and Z+jets background 
events. We generate both SM V - A and V + A 
Wtb couplings, and reweight events to model a given 
W boson helicity state. We use the CTEQ6L1 parton 



where Mw is the mass of the W boson, q v is its 
four-momentum, Vt& is the Cabibbo-Kobayashi-Maskawa 
matrix element Q, and P L = (l-7 5 )/2 (P R = (l+7 5 )/2) 
is the left-handed (right-handed) projection operator. 
In the SM, the Wtb coupling is purely left-handed, 
and the values of the coupling form factors are /f r* 
1, f 2 = fi = = 0. We assume real coupling form 
factors, implying CP conservation, and a spin-i top 
quark which decays predominantly to Wb. Indirect 
constraints on the magnitude of the right-handed vector 
coupling and tensor couplings exist from measurements 
of the b — > S7 branching fraction 0]. While those limits 
are tighter than the direct limits presented here, they 
also include assumptions that are not required here, for 
example the absence of other sources of new physics 
coupling to the b quark. 

We search for non-SM values of the couplings using 
?s 1 fb" 1 of data collected by the DO experiment [5[ 
at the Fermilab Tevatron pp collider between 2002 and 
2006 (Run II). Variations in the coupling form factors 
would mainly manifest themselves in two distinct ways 
at DO: by changing the rate and kinematic distri- 
butions of electroweak single top quark production, and 
by altering the fractions of W bosons from top quark 
decays produced in each of the three possible helicity 
states. In this Letter, we combine information from 
our measurement of the W boson helicity fractions in 
ti events Q with information from single top quark 
production. We have previously set direct limits on 
anomalous top quark coupling form factors based solely 
on the single top quark final state 0]. Here we set 
substantially tighter limits on the effective top quark 
couplings using the general framework given in Ref. Q. 
This is the first such combination of all applicable DO 
Run II top quark measurements to limit anomalous top 
quark coupling form factors. 

We follow the approach adopted in Ref. Q and 
investigate one pair of coupling form factors at a time 
out of the full set of form factors (//", fi, f 2 , and 
f 2 . For each pair under investigation we assume that 
the other two have the SM values. We consider three 



distribution functions ll| and set the top quark mass 
to 172.5 GeV. The response of the DO detector to the 
MC events is simulated using GEANT [l^]. We model 
the background from multijet production where a jet is 
misidentificd as an isolated electron or muon using events 
from data containing lepton candidates which pass all 
of the lepton identification requirements except one but 
otherwise resemble the signal events. We use MC to 
model other small backgrounds (diboson and single top 
quark production) . 

We select events with a multivariate likelihood 
discriminant that uses both kinematic and 6-lifctimc 
information to distinguish ti events from background and 
obtain a sample of 288 ^+jets (75 dilepton) events with 
an expected background contribution of 54 ± 7 (17 ± 4) 
events. 

A binned maximum likelihood fit compares the cos#* 
distribution of the selected events to the expectations for 
each W boson helicity state plus background. We vary 
both the longitudinal and right-handed helicity fractions 
fo and f + in the fit and find the relative likelihood 
of any set of helicity fractions being consistent with 
the data. In the previous W helicity publication, we 
expressed the likelihood in terms of the helicity fractions 
and used a prior that was flat in fo and /+ Q. Here, we 
instead express these relative likelihoods in terms of the 
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anomalous Wtb coupling form factors squared using the 
relationships given in Ref. The resulting likelihood 
distributions are shown in the left column of Fig.[TJ They 
show that the W helicity measurement only constrains 
ratios of the coupling form factors. 

We can constrain both the ratios and the magnitudes of 
the form factors in the single top analysis. The dominant 
modes for single top quark production at the Tevatron 
are the s-channel production and decay of a virtual 
W boson and the f-channel exchange of a If boson. 
Evidence for production of single top quarks has been 



reported by the DO and CDF collaborations [13|, Il4|. 
Both the cross section and the angular correlations of 
the final state objects are modified in the presence of 
anomalous couplings. The total cross section for SM 
single top quark production at a top quark mass of 
172.5 GeV is predicted to be 3.15 ± 0.3 pb [ij]. For this 
analysis, we assume that single top quarks are produced 
exclusively through W boson exchange and that the Wtb 
vertex dominates top quark production and decay. 

We look for single top quark production in events with 
one lepton [electron (p T > 15 GeV) or muon (p T > 
18 GeV)] and $ T > 15 GeV. We select a sample that 
is statistically independent of the W helicity analysis 
by asking for two or three jets with px > 15 GeV, of 
which one should have pr > 25 GeV. We also require 
at least one of the jets to be identified as originating 
from a b hadron by a 6-tagging algorithm. Details of the 
selection criteria and background modeling are given in 
Ref. [H . 

We model the single top quark signal using the 
COMPHEP-SINGLETOP MC event generator [l6| where 
anomalous Wtb couplings are considered in both the 
production and decay of the top quark. The background 
modeling for the single top analysis utilizes the same 
samples as the W helicity analysis for VK+jets and 
multijct backgrounds. The ti background in the single 
top quark sample is small and is modeled by simulated 
SM tt events. It is normalized to the theoretical cross 
section 

The selection efficiencies for single top quark signals 
with different Wtb couplings are approximately (1-2)% 
for events with one b tag and less than 1% for events 
with two b tags. We select 1152 events, which we expect 
to contain 56 ± 12 SM single top quark events. We use 
boosted decision trees 



It 



19j to extract single top quark 
events from the large background. 

Systematic uncertainties in the signal and background 
models are described in detail in Refs. [f| and We 
take all systematic uncertainties and their correlations 
into account. Systematic uncertainties in the W 
boson helicity measurement arise from finite MC 
statistics and uncertainties on the top quark mass, 
jet energy calibration, and MC models of signal and 
background. Variations in these parameters can change 
the measurement in two ways: by altering the estimate 



of the background in the final sample (i.e., if the final 
selection efficiency changes) and by modifying the shape 
of the cos0* templates. Systematic uncertainties in the 
single top analysis arise from the W +jets normalization, 
the W^+jets flavor composition estimate, and the top 
quark pair background modeling. 

Most of the systematic uncertainties are taken to be 
100% correlated between the two analyses. Systematic 
uncertainties that affect only the W helicity analysis arc 
MC statistics and MC background model. Systematic 
uncertainties arising from the luminosity measurement 
affect only the single top analysis. 

We use a Bayesian statistical analysis [2(| to combine 
the W helicity result with the single top anomalous 
coupling result. The likelihood result from the W helicity 
analysis is used as a prior to the single top anomalous 
coupling analysis. 

For any pair of values of the two coupling form factors 
under consideration, we compare the boosted decision 
tree output for the data with the sum of backgrounds and 
the two signals. In the scenario where and f% are non- 
zero, the two amplitudes interfere, which we take into 
account by using a superposition of three signal samples: 
one with only left-handed vector couplings; one with only 
left-handed tensor couplings; and one with both coupling 
form factors set to one, containing the interference term. 
We then compute a likelihood as a product over all 
separate analysis channels. We assume Poisson distri- 
butions for the observed counts and use multivariate 
Gaussian distributions to model the uncertainties on 
the combined signal acceptance and background yields, 
including correlations. The uncertainties are evaluated 
through MC integration. We generate an ensemble of 
5000 samples, each with a different shift in the various 
systematic uncertainties, and compute the Bayesian 
posterior for each sample. The final posterior is then 
the ensemble average of all individual posteriors. 

The two-dimensional posterior probability density is 
computed as a function of |/f| 2 and \fx\ 2 , where fx is 
fi, /£, or /f . These probability distributions arc shown 
in Fig.[TJ In all three scenarios we measure approximately 
zero for the anomalous coupling form factors and favor 
the left-handed vector hypothesis over the alternative 
hypothesis. We compute 95% Confidence Level (C.L.) 
upper limits on these form factors by integrating out 
the left-handed vector coupling form factor to get a one- 
dimensional posterior probability density. The measured 
values are given in Table |TJ 

In comparison, the limits at 95% C.L. without the 
W helicity constraints are \f?\ 2 < 1.83, |/ 2 L | 2 < 0.52, and 
IJ.fl 2 < 0.24. The kinematic distributions of the and 
fi' single top quark samples are similar enough that the 
single top anomalous coupling analysis in this scenario 
is mainly sensitive to the total cross section. Hence, the 
W helicity analysis improves the \f^\ 2 limit significantly. 
Conversely, it does not add much information to the 
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right-tensor coupling limit where most of the sensitivity 
is provided by the single top anomalous coupling analysis. 
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FIG. 1: W helicity prior (a, c, e) and final posterior density 
(b, d, f) for right- vs left-handed vector coupling (a and b), 
left-handed tensor vs left-handed vector coupling (c and d), 
and right-handed tensor vs left-handed vector coupling (e and 
f). The W helicity prior is normalized to a peak value of one 
and shown as equally spaced contours between zero and one. 
The posterior density is shown as contours of equal probability 
density. 

In summary, we have presented the first study of Wtb 
couplings that combines W helicity measurements in top 
quark decay with anomalous couplings searches in the 
single top quark final state, thus using all applicable 
top quark measurements by DO. We find consistency 
with the SM and set 95% C.L. limits on anomalous Wtb 
couplings. Our limits represent significant improvements 
over previous results, and rule out a right-handed top 
quark vector coupling form factor of magnitude one for 
the first time. 
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TABLE I: Measured values with uncertainties and upper 
limits at the 95% C.L. for Wtb couplings in three different 
scenarios. 
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